Skip to content

Run the chat-app Web smoke in Safari and Firefox and add a WebGPU generation E2E (#1025) - #2002

Merged
leehack merged 2 commits into
mainfrom
lane-1025-browser-harness-r2
Nov 14, 2026
Merged

leehack merged 2 commits into
mainfrom
lane-1025-browser-harness-r2

Conversation

@leehack

@leehack leehack commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • run_local_e2e.dart --scenario chat-app-web-real-model-smoke --browser safari|firefox serves the repo with --coep require-corp and drives the smoke through safaridriver or geckodriver (FIREFOX_EXECUTABLE, GECKODRIVER).
  • --backend webgpu passes --backend-index 0 --gpu-layers 99 (default stays CPU Wasm).
  • The Dart web scenarios reject --browser safari|firefox: package:test pages are never cross-origin isolated.
  • New web-webgpu-generation scenario and test/e2e/webgpu/webgpu_generation_e2e_test.dart (local-only): full-offload WebGPU text vs CPU Wasm, direct and worker.
  • Firefox injects the init script as a page <script> because geckodriver's execute sandbox hides page globals.
  • --backend webgpu now also passes --expect-gpu-active true (a CPU fallback fails) and, with --backend, only cpu or webgpu are accepted by this scenario (stricter allowlist); --backend webgpu is rejected on other scenarios. Safari and Firefox run Gate Web ASR runtime and microphone lifecycle in the harness (#329) #1999's runtime probe.
  • Firefox page-state reads run in the page world, like the init script.
  • Supersedes Run the chat-app Web smoke in Safari and Firefox and add a WebGPU generation E2E (#1025) #2000 (rebased on Gate Web ASR runtime and microphone lifecycle in the harness (#329) #1999; the head-update tool cannot fast-forward a rebase).
  • Part of #1025, plan #1995. Support wording lands with the repin PR.

Production-readiness scope

  • User-facing scope: maintainer test harness only; no lib/ change.
  • Supported platforms/paths: macOS Safari 27, Firefox via geckodriver, Chrome WebGPU.
  • Unsupported or intentionally unavailable paths: Safari/Firefox on Dart web scenarios exit 64 with the isolation reason; tool-call, speech and mmproj checks in Safari/Firefox exit 2.
  • Out of scope / follow-ups: Firefox real run (see Matrix Evidence); coordinates with #1999, which edits the same four files in separate hunks.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. (N/A: no public API; matrix doc updated; changelog and support docs belong to the repin PR)
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Public capability changes update the validation coverage registry. (N/A: none)
  • Security/privacy review completed: no secrets leak.
  • Follow-up work tracked, or "None" above.

PR type guidance

  • Feature PR: N/A, test tooling only.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • dart test -p vm test/unit/tooling/ (1830 passed); full -j 1 VM and Chrome suites left to CI
  • coverage-lib: N/A, no lib/ change
  • Other: mutation check (dropping --coep require-corp fails the dry-run and matrix tests); real runs below

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
web-real-model-smoke-safari chat app, plain chat, require-corp Safari 27.0, macOS, Qwen3.5-0.8B-Q4_K_M, CPU Wasm PASS 20.1 s, reply "2+2 equals 4."
web-real-model-smoke-safari same, --backend webgpu (requests 99 layers; Qwen3.5-0.8B is capped at 2, so partial offload) Safari 27.0, Apple WebGPU, same model PASS (before the GPU-active assertion existed) 21.5 s, runtime label WEBGPU; superseded, see open item
web-real-model-smoke-safari --backend webgpu --expect-gpu-active true, stories15M, --allow-any-response Safari 27 NOT RE-RECORDED host screen locked (page hidden, 0 frames); rerun when unlocked
web-real-model-smoke-firefox chat app, plain chat Firefox 160 UNQUALIFIED, not yet run scratch Firefox 160 and geckodriver were not on this machine and the download was not permitted; covered by the scripted-driver unit test only
web-webgpu-generation direct and worker, 8 and 32 tokens Chrome WebGPU (Apple metal-3), stories15M PASS WebGPU text equals CPU text in both runtimes

Review Notes

  • Independent review status: pending (high-risk)
  • CI status / head SHA: pending, pending

High-risk regression review

  • Classification: high-risk (regressionPolicy)
  • Implementation task: this PR
  • Independent blocking QA task: pending
  • Exact head / current base: pending
  • Production-branch deletion, bypass, or miswire proof: N/A
  • Affected-family real-model/artifact evidence: Qwen3.5-0.8B (Safari), stories15M (Chrome generation)
  • Explicit unavailable-family or other N/A evidence: Firefox real run
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

…eration E2E (#1025)

--browser safari|firefox serves require-corp and drives the real-model smoke
through safaridriver or geckodriver; --backend webgpu offloads all layers.
The Dart web scenarios reject both browsers: package:test pages are never
cross-origin isolated. web-webgpu-generation compares full-offload WebGPU
text with CPU Wasm in direct and worker runtimes.
…the page world (#1025)

Safari and Firefox now run the GPU-active probe; a CPU-labelled run fails.
Firefox page-state reads run as page <script> elements like the init script.
The Firefox matrix row is marked unqualified until a real run passes.
@leehack
leehack marked this pull request as ready for review November 14, 2026 03:00
@leehack
leehack merged commit cde82af into main Nov 14, 2026
43 checks passed
@leehack
leehack deleted the lane-1025-browser-harness-r2 branch November 14, 2026 03:00
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-2002.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant